Distributed Stochastic Optimization with Large Delays

The recent surge of breakthroughs in machine learning and artificial intelligence has sparked renewed interest large-scale stochastic optimization problems that are universally considered hard. One the most widely used methods for solving such is distributed asynchronous gradient descent (DASGD), a family algorithms result from parallelizing on computing architectures (possibly) asychronously. However, key obstacle efficient implementation DASGD issue delays: when node contributes update, global model parameter may have already been updated by other nodes several times over, thereby rendering this information stale. These delays can quickly add up if computational throughput saturated, so convergence be compromised presence large delays. Our first contribution that, carefully tuning algorithm’s step size, to critical set still achieved mean square, even grow unbounded at polynomial rate. We also establish finer results broad class structured (called variationally coherent), where we show converges optimum with probability one under same delay assumptions. Together, these contribute landscape nonconvex offering state-of-the-art theoretical guarantees providing insights algorithm design.